Papers with neural sequence models
Grokking of Hierarchical Structure in Vanilla Transformers (2023.acl-short)
Copied to clipboard
| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
Sub-event detection from twitter streams as a sequence labeling problem (N19-1)
Copied to clipboard
| Challenge: | Existing methods for sub-event detection do not account for sequential nature of social media streams. |
| Approach: | They propose to use a neural sequence architecture that explicitly accounts for the chronological order of posts to improve sub-event detection. |
| Outcome: | The proposed method outperforms a graph-based state-of-the-art method for binary sub-event detection (2.7% micro-F1 improvement) it also outperformed a recurrent neural network model on the posts sequence level for labeled sub- events (2.4% bin-level improvement). |
Learning to Substitute Spans towards Improving Compositional Generalization (2023.acl-long)
Copied to clipboard
| Challenge: | despite the rising prevalence of neural sequence models, there is a deficiency in compositional generalization. |
| Approach: | They propose a compositional augmentation strategy that enables multi-grained composition of substructures in the whole training set. |
| Outcome: | The proposed strategy outperforms existing strategies on three compositional generalization benchmarks. |
On NMT Search Errors and Model Errors: Cat Got Your Tongue? (D19-1)
Copied to clipboard
| Challenge: | We show that at the root of the problem of empty translations lies an inherent bias towards shorter translations. |
| Approach: | They propose an exact inference procedure for neural sequence models based on beam search and depth-first search. |
| Outcome: | The proposed procedure finds that beam search fails to find the best model scores . the results show that the model often prefers an empty translation . |
Sequence Models for Computational Etymology of Borrowings (2021.findings-acl)
Copied to clipboard
| Challenge: | a computational model of word borrowing can be useful for lexicon expansion and language preservation. |
| Approach: | They propose to use neural sequence models to model word borrowings from a donor word to an incorporated word. |
| Outcome: | The proposed model beats baseline models in both directions, with the quantity of data strongly influencing performance. |
Ab Antiquo: Neural Proto-language Reconstruction (2021.naacl-main)
Copied to clipboard
| Challenge: | Historical linguists have identified regularities in the process of historic sound change. |
| Approach: | They propose a method to reconstruct proto-words based on cognates in daughter languages . they use a dataset of 8,000 comparative entries to analyze phonological changes . |
| Outcome: | The proposed method outperforms conventional methods in a proto-word reconstruction task. |
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)
Copied to clipboard
| Challenge: | Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks. |
| Approach: | They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features . |
| Outcome: | The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias. |
Neural Grammatical Error Correction with Finite State Transducers (N19-1)
Copied to clipboard
| Challenge: | Language model based GEC (LM-GEC) is a promising alternative to SMT and neural sequence-to-sequence models. |
| Approach: | They propose to use finite state transducers to improve LM-GEC by rescoring with neural language models. |
| Outcome: | The proposed model outperforms the best published results on the CoNLL-2014 test set and achieves far better relative improvements over the baselines. |
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. |
| Approach: | They propose to use a recurrent language model to address inconsistency in decoding algorithms that are inconsistent despite the fact that recursive language models are trained to produce sequences of finite length. |
| Outcome: | The proposed methods prevent inconsistency in the proposed models. |
Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models (2022.acl-long)
Copied to clipboard
| Challenge: | In many natural language processing tasks the same input can have multiple possible outputs. |
| Approach: | They propose a novel exact n-best search algorithm for neural sequence models to measure sentence-level uncertainty by computing the degree of overlap between references from two different NLP tasks. |
| Outcome: | The proposed algorithm overly spreads the probability mass for uncertain tasks and sentences. |